Databricks Architect [Python and SQL]

Project Description:

We are seeking a highly experienced Databricks Architect to lead the design, implementation, and optimization of enterprise-grade data platforms and AI-enabled workloads for a global food corporation. This role requires a hands-on technical leader who can translate business and data requirements into scalable Databricks-based solutions while ensuring robust governance, performance, and operational resilience. The ideal candidate will combine deep Databricks expertise with strong architecture leadership and the ability to operate within a managed service delivery model.

Responsibilities

:Lead the architecture, design, and implementation of Databricks-based data platforms, pipelines, and AI/ML workloads
.Define scalable lakehouse patterns using core Databricks capabilities such as Apache Spark, Delta Lake, Databricks SQL, MLflow, and governance components
.Design robust batch and streaming data pipelines for high-volume, business-critical workloads
.Establish architecture standards for performance optimization, workload orchestration, reliability, observability, and cost control
.Translate business and technical requirements into solution blueprints, reference architectures, and implementation roadmaps
.Collaborate with data engineers, analysts, data scientists, DevOps teams, and business stakeholders to deliver end-to-end solutions
.Ensure strong data governance, security, and compliance practices across the Databricks environment
.Support AI and advanced analytics use cases by enabling reliable feature engineering, model lifecycle practices, and production-ready data foundations
.Provide technical leadership during solution delivery, troubleshooting, and optimization of existing workloads
.Act as a trusted advisor to client stakeholders, offering recommendations on architecture decisions, risks, trade-offs, and delivery priorities

.
Mandatory Skills Descriptio

n:Deep hands-on expertise with the Databricks platform, including architecture, workspace design, cluster strategy, jobs orchestration, and platform optimizatio
n.Strong command of Apache Spark and distributed data processing concepts, including performance tuning and optimization for complex data engineering workload
s.Proven experience designing and delivering enterprise data solutions using Delta Lake and lakehouse architecture principle
s.Strong proficiency in Python and SQL; Scala is an advantag
e.Demonstrated capability in building and optimizing ETL/ELT pipelines, data models, and large-scale ingestion framework
s.Experience supporting AI/ML workloads on Databricks, including MLflow, model lifecycle considerations, and production-ready data preparatio
n.Solid knowledge of cloud-native architecture patterns on AWS, Azure, or GCP, including storage, networking, identity, and security integratio
n.Experience with data governance, access control, lineage, and compliance frameworks in enterprise environment
s.Familiarity with CI/CD, infrastructure-as-code, monitoring, and operational best practices for data platform
s.Ability to engage in deep technical problem-solving and contribute immediately to complex delivery scenarios with minimal ramp-up tim

e.
Nice-to-Have Skills Descripti

on:Databricks certifications relevant to data engineering, machine learning, or platform architectu
re.Experience in large enterprise or global delivery environments with complex stakeholder landscap
es.Background in consumer goods, manufacturing, supply chain, or similarly data-intensive industri
es.Experience with real-time processing, orchestration frameworks, and integration with enterprise data ecosyste
ms.Strong communication skills with the ability to explain architecture decisions to both technical and non-technical stakeholde

rs.